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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address -- 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

Responsive to communication(s) filed on 31 October 2003 . 
2a)£3 This action is FINAL. 2b)D This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) K Claim(s) 1 ,3-18.20-35.37-52,54-66.68-80 and 82-94 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) [EI Claim(s) 1. 3-18. 20-35. 37-52, 54-66. 68-80 and 82-94 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) Q Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10)Q The drawing(s) filed on is/are: a)Q accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
11 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 
Priority under 35 U.S.C. §§119 and 120 

12) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

13) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application) 

since a specific reference was included in the first sentence of the specification or in an Application Data Sheet. 
37 CFR 1.78. 

a) D The translation of the foreign language provisional application has been received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 since a specific 

reference was included in the first sentence of the specification or in an Application Data Sheet. 37 CFR 1 .78. 



Attachment(s) 

1) □ Notice of References Cited (PTO-892) 

2) d Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) Q Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 



4) □ Interview Summary (PTO-413) Paper No(s). 

5) Q Notice of Informal Patent Application (PTO-152) 

6) □ Other: 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 11-03) 



Office Action Summary 



Part of Paper No. 17 




Application/Control Number: 09/395,993 
Art Unit: 2623 



Page 2 



DETAILED ACTION 



Response to Amendment 



1 . Applicant's amendment filed on October 31 , 2003, has been entered and made 
of record. 

2. Applicant's arguments regarding Claims 1,18, 35, 52, 66 and 80 have been fully 
considered but they are not persuasive. 

Applicant argues that prior art of record (Modestino et al) do not disclose, 
"analyzing a region adjacency graph to identify predetermined patterns of the semantic 



The Examiner disagrees and indicates that Modestino et al clearly disclose this 
limitation (Abstract; Figure 1; Pages 608-609, Section lll-B. The image in Figure 1a is 
segmented into homogeneous nodes or regions, R = {Ri, Rn) as depicted in 

Figure 1(c). Section NIB describes predetermined patterns of the semantic labels; e.g., 
the second type of clique basis functions for spatial constraints, "a car should be on the 
road" or "a car should never be in the sky.). 

Applicant further argues that Modestino do not teach or suggest assigning a 
predetermined stereotype to digital image according to each with a plurality of semantic 
labels in the labeled region adjacency graph. 

The Examiner disagrees and indicates as discussed above, Modestino et al 
analyzes predetermined patterns of the semantic labels in the labeled region adjacency 
graph. In Page 606, Introduction, Modestino et al clearly indicates "In high-level 
processing, image domain knowledge is used to assign object label to the primitives, 



labels 
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thereby providing a semantic description of the image. The joint assignment of object 
labels for all the image primitives provides a stereotype interpretation of the 
image. 

This means a stereotype (high-level interpretation) is associated with the image 
semantic label. Modestino et al teachings clearly disclose this concept (Figures 1(a)- 
1(c), 5(a)-5(d) and 6(a)-6(d); Tables l-IV; Page 606, Section I, Introduction, Paragraphs 
1-3; Pages 610-613, Sections V-VII. Modestino et al disclose a Markov Random Field 
(MRF) model-based approach image interpretation and classification by performing 
knowledge-based high-level processing via assigning higher-level expressions 
(Stereotypes) to the classified objects. The region adjacency graph is classified based 
on higher-level expression or stereotypes (e.g., rural road scenes comprising road, field, 
car and sky). "Two regions" and "three regions" categories depicted in Table I in 
association of Figures 1(a)-1(c) are stereotype or higher-level expressions.). 

Furthermore, the teachings of secondary prior art of record, Jain et al (US 
6,360,234), relied upon for the rejection of Claims 13-16, 47-51 and 30-33 in previous 
Office Action, also disclose assigning stereotypes to digital images based on the 
features and characteristics of the labeled segments of the digital image. The 
Examiner, for further emphasis, includes the same basis for rejection of Claims 13-16, 
47-51 and 30-33, in the rejection of amended Claim 1 and analogous Claims. 



3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 



Claim Rejections - 35 USC § 103 
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(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 1, 3-10, 13-16, 18, 20-27, 30-33, 35, 37-44, 47-50, 52, 54-61, 66, 68-75, 
80, 82-89 and 94 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Modestino et al (IEEE Paper ISBN: 0162-8828; A Markov random Field Model-Based 
Approach to Image Interpretation) in view of Jain et al (US 6,360,234). 

Regarding Claim 1 , Modestino et al disclose a method of classifying a digital 
image, said method comprising the steps of: 

segmenting the digital image into substantially homogeneous regions (Figure 1; 
Pages 607-608, Section ll-B. The image in Figure 1a is segmented into homogeneous 
nodes or regions, R = {Ri, R 2 , R n ) as depicted in Figure 1(c).); 

processing the regions to provide a labeled region adjacency graph (Figure 1 ; 
Section II) for the digital image, the region adjacency graph representing adjacencies 
between the regions of the digital image, at least one of the regions of the labeled 
region adjacency graph being associated with one of a plurality of predetermined 
semantic labels (Figure 1 ; Page 607, Column 2, Section MB, first Paragraph. G = {R, £} 
is the region adjacency graph comprising of the set of nodes R = {Ri, R2, Rn} and 
the set of edges E connecting the regions. The labels assigned to the segmented 
regions. L = {L 1t L2, L M } is the set of all predetermined semantic labels.); 

analyzing the labeled region adjacency graph to identify one or more 
predetermined patterns of the semantic labels in the labeled region adjacency graph 
(Abstract; Figure 1; Pages 608-609, Section lll-B. The image in Figure 1a is segmented 
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into homogeneous nodes or regions, R = f?2, Rn) as depicted in Figure 1(c). 
Section 1MB describes predetermined patterns of the semantic labels; e.g., the second 
type of clique basis functions for spatial constraints, "a car should be on the road" or "a 
car should never be in the sky. Predetermined patterns of the semantic labels are 
identified as shown in Tables l-IV, i.e., possible combinations and impossible 
combinations.); and 

assigning one or more of a plurality of predetermined stereotypes (based on 
reasonable broad interpretation of stereotype) to the digital image according to each 
identified predetermined pattern of the semantic labels in the labeled region adjacency 
graph, wherein each of the predetermined stereotypes corresponds to at least one of 
the predetermined patterns such that the assigned stereotype represents a 
classification of the digital image based on each predetermined pattern identified in the 
labeled region adjacency graph (Figures 1, 5(a)-5(d), 6(a)-6(c); Page 606, Introduction; 
Pages 608-609, Section lll-B; Tables l-IV. The region adjacency graph is classified 
based on higher-level expression or stereotypes (e.g., rural road scenes comprising 
road, field, car and sky. "Two regions" and "three regions" categories depicted in Table I 
in association of Figures 1(a)-1(c) are components of stereotypes or higher-level 
expressions. The joint assignment of object labels for all the image primitives provides 
a stereotype for interpretation of the image. The digital image is classified based on 
the identified patterns (features) of the regions as depicted in Figures 1, 5(a)-5(d), 6(a)- 
6(c) and Tables l-IV.). 
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Furthermore, Jain et al also disclose providing metadata associated with the 
digital image, wherein the metadata includes the stereotypes of the digital image 
(Narrower interpretation of stereotype; Figures 16 and 17; Column 13, Lines 52-67, 
Column 14, Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al assigning one or more of a plurality of predetermined stereotypes to the digital 
image according to each identified predetermined pattern of the semantic labels in the 
labeled region adjacency graph, wherein each of the predetermined stereotypes 
corresponds to at least one of the predetermined patterns such that the assigned 
stereotype represents a classification of the digital image based on each predetermined 
pattern identified in the labeled region adjacency graph because it will increase the 
accuracy and reliability of the classification system. It will intelligently interpret digital 
images based on the high and low-level characteristics, features and contents of the 
images. 

Regarding Claim 3, Modestino et al further disclose the method according to 
Claim 1, wherein identification of the predetermined pattern is based on a size of one or 
more regions of the digital image (Section III, first paragraph; Section NIB; Tables l-IV. 
Classification is based on the areas and the boundary lengths of the regions identified 
by the RAG nodes.). 

Regarding Claim 4, Modestino et al further disclose the method according to 
Claim 1 , wherein identification of the predetermined pattern is based on an adjacency of 
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the regions (Figure 1; Pages 608-609, Section III-B; Figures 1 and 6(b). Region 
adjacency graphs are inherently generated based on the spatial dependence of objects 
in the relative proximity of each other as depicted in Figure 1(c).). 

Regarding Claim 5, Modestino et al further disclose the method according to 
Claim 1, wherein identification of the predetermined pattern is based on semantic label 
content of the region adjacency graph (Figure 1; Pages 608-609, Section III). 

Regarding Claim 6, Modestino et al further disclose the method according to 
Claim 1, wherein identification of the predetermined pattern is based on a mean color of 
one or more regions of the digital image (Figure 1; Pages 608-609, Section III; Page 
613, Table lll(a), Average Gray Level of the regions). 

Regarding Claim 7, Jain et al further disclose the method according to Claim 1 , 
wherein the plurality of stereotypes are stored in an association lookup table (Figures 6, 
9, 16 and 17; Column 6, Lines 29-47; Column 13, Lines 52-67, Column 14, Lines 1-31). 

Regarding Claim 8, Jain et al further disclose the method according to Claim 1 , 
wherein the stereotypes are represented in a hierarchal arrangement (Figures 7, 9 and 
17; Column 6, Lines 29-47; Column 13, Lines 52-67, Column 14, Lines 1-31). 

Regarding Claim 9, Jain et al further disclose the method according to Claim 7, 
wherein each of the stereotypes has a hierarchical path arrangement (Figures 6, 7, 9 
and 17; Column 6, Lines 29-47; Column 13, Lines 52-67, Column 14, Lines 1-31). 

Regarding Claim 10, Modestino et al disclose the method according to Claim 1, 
wherein the region adjacency graph is provided by analyzing contextual data associated 
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with one or more regions of the digital image (Abstract; Page 606, Introduction; Figure 
1 , Page 607-609, Sections MB and 3, first Paragraph). 

Regarding Claim 13, Modestino et al do not disclose the method according to 
Claim 1, further comprising the step of providing metadata associated with the digital 
image, wherein the metadata includes the stereotypes of the digital image. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image comprising the step of providing metadata associated with 
the digital image, wherein the metadata includes the stereotypes of the digital image 
(Figures 16 and 17; Column 13, Lines 52-67, Column 14, Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al to provide metadata associated with the digital image, wherein the metadata 
includes the stereotypes of the digital image because it will improve image appearance 
and enhance image classification. It will accurately classify and intelligently extract 
information, termed metadata, about the contents of video stream in real time (Jain et 
al, Column 1, Lines 46-49) 

Regarding Claim 14, Jain et al further disclose the method according to Claim 13, 
wherein the metadata includes a hierarchical path associated with the respective 
stereotype of each digital image (Figures 9, 15-17; Table 1; Column 8, Lines 22-61). 

Regarding Claim 15, Jain et al further disclose the method according to Claim 14, 
wherein the hierarchical path is stored with a respective stereotype as a metadata 
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object which is associated with a respective image object (Figure 6; Column 6, Lines 
29-38). 

Regarding Claim 16, Jain et al further disclose the method according to Claim 14, 
wherein the hierarchical path is stored as a referenced lookup table (Figure 7). 

With regards to Claims 18, 35, 52, 66, 80 and 94, arguments analogous to those 
presented for Claim 1 are applicable to Claims 18, 35, 52, 66, 80 and 94. Concerning 
Claims 52, 66 and 80, Modestino et al further disclose providing a set of labeled regions 
(Abstract Lines 5-10; Page 607, Column 2, Section MB, second paragraph, set of labels 
L = {Li, L 2 , .... L M }). Markov Random Field (MRF) model-based approach segments the 
image into a collection of disjoint regions that form the nodes of an adjacency graph. 
Once the adjacency graph has been determined, image classification will be achieved 
through assigning object labels to the segmented regions using domain knowledge, 
extracted feature measurements, and spatial relationship between the various regions 
(Abstract, Lines 5-10). 

With regards to Claims 20, 37, 56, 70 and 84, arguments analogous to those 
presented for Claim 3 are applicable to Claims 20, 37, 56, 70 and 84. 

With regards to Claims 21 , 38, 55, 69 and 83, arguments analogous to those 
presented for Claim 4 are applicable to Claims 21 , 38, 55, 69 and 83. 

With regards to Claims 22, 39, 54, 68 and 82, arguments analogous to those 
presented for Claim 5 are applicable to Claims 22, 39, 54, 68 and 82. 




Application/Control Number: 09/395,993 Page 10 

Art Unit: 2623 

With regards to Claims 23, 40, 57, 58, 71 , 72, 85 and 86, arguments analogous 
to those presented for Claim 6 are applicable to Claims 23, 40, 57, 58, 71 , 72, 85 and 
86. 

With regards to Claims 24, 41 , 59, 73 and 87, arguments analogous to those 
presented for Claim 7 are applicable to Claims 24, 41 , 59, 73 and 87. 

With regards to Claims 25, 42, 60, 74 and 88, arguments analogous to those 
presented for Claim 8 are applicable to Claims 25, 42, 60, 74 and 88. 

With regards to Claims 26, 43, 61, 75 and 89, arguments analogous to those 
presented for Claim 9 are applicable to Claims 26, 43, 61 , 75 and 89. 

With regards to Claims 27 and 44, arguments analogous to those presented for 
Claim 10 are applicable to Claims 27 and 44. 

Regarding Claim 30, Modestino et al do not disclose the method according to 
Claim 18 further comprising the step of providing metadata associated with the digital 
image, wherein the metadata includes the stereotypes of the digital image. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the digital image, wherein the metadata includes stereotypes of the 
digital image (Figures 16 and 17; Column 13, Lines 52-67, Column 14, Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al to provide metadata associated with the digital image, wherein the metadata 
includes stereotypes of the digital image because it will accurately classify and 
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intelligently extract information, termed metadata, about the contents of video stream in 
real time (Jain et al, Column 1 , Lines 46-49). 

With regards to Claim 31, arguments analogous to those presented for Claim 14 
are applicable to Claim 31 . 

With regards to Claim 32, arguments analogous to those presented for Claim 15 
are applicable to Claim 32. 

With regards to Claim 33, arguments analogous to those presented for Claim 16 
are applicable to Claim 33. 

With regards to Claim 47, arguments analogous to those presented for Claim 13 
are applicable to Claim 47. 

With regards to Claim 48, arguments analogous to those presented for Claim 14 
are applicable to Claim 48. 

With regards to Claim 49, arguments analogous to those presented for Claim 15 
are applicable to Claim 49. 

With regards to Claim 50, arguments analogous to those presented for Claim 16 
are applicable to Claim 50. 

5. Claims 1 1 , 1 7, 28, 34, 51 , 45, 62-65, 76-79 and 90-93 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Modestino et al further in view of Jain et al 
(US 6,360,234) and Li et al (U.S. 5,930,783). 

Regarding Claim 11, Modestino et al and Jain et al do not explicitly disclose the 
method according to Claim 10, wherein the contextual data comprises information 
generated by one or more separate sources of the information. 
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Li et al disclose a semantic and cognition based image retrieval methodology 
comprising analyzing contextual data generated by one or more separate sources of 
information (Figure 1B, Semantic-based Query, Cognition-based Query; Column 12, 
Lines 30-40). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Jain et al combination according to 
the teachings of Li et al to analyze contextual data generated by one or more separate 
sources of information because it will expand versatility of the image segmentation and 
classification. It will utilize the advantages of image retrieval based on both image 
semantics and visual examples of the image (Li et al, Column 3, Lines 27-35). 

Regarding Claim 17, Modestino et al and Jain et al do not explicitly disclose the 
method according to Claim 1 , wherein said digital image is stored in a database of 
digital images and wherein said classification can be used to retrieve said digital image 
from said database. 

Li et al disclose a semantic and cognition based image retrieval methodology 
wherein the digital image is stored in a database of digital images and wherein the 
classification can be used to retrieve said digital image from the database (Column 4, 
Lines 32-50). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Jain et al combination according to 
the teachings of Li et al to use classification to retrieve the digital images stored in a 
database because it will utilize a plurality of information sources for image retrieval 
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including semantic-based, cognition-based and scene-based queries thereby 
eliminating weakness of the individual approaches (Li et al; Column 3, Lines 27-35). 

With regards to Claims 28 and 45, arguments analogous to those presented for 
Claim 1 1 are applicable to Claims 28 and 45. 

With regards to Claim 34, arguments analogous to those presented for Claim 17 
are applicable to Claim 34. 

With regards to Claim 51 , arguments analogous to those presented for Claim 17 
are applicable to Claim 51 . 

With regards to Claims 63, 77 and 91, arguments analogous to those presented 
for Claim 17 are applicable to Claims 63, 77 and 91. Li et al retrieve the digital image 
by using a keyword representing a stereotype (Column 12, Lines 30-50). 

With regards to Claims 64, 78 and 92, arguments analogous to those presented 
for Claim 17 are applicable to Claims 64, 78 and 92. Modestino et al, Jain et al and Li 
et al do not explicitly retrieve the digital image by using an icon to represent a 
stereotype. Utilizing an icon to represent a keyword is extremely well known in the art 
(Official Notice). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al, Jain et al and Li et al combination to 
Utilize an icon for representing a keyword because icons are a significant factor in the 
user- friendliness of graphical user interface that serve as visual mnemonics to allow the 
user to control certain computer actions without having to remember commands or type 
them at the keyboard. 
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With regards to Claims 62, 76 and 90, arguments analogous to those 
represented for Claim 64 concerning utilizing an icon is applicable to claims 62, 76 and 
90. 

With regards to Claims 65, 79 and 93, arguments analogous to those presented 
for Claim 17 are applicable to Claims 65, 79 and 93. Li et al retrieve the digital image 
by using a keyword representing a generalization of a stereotype (Column 12, Lines 30- 
50). 

Regarding Claim 12, neither Modestino et al nor Li et al disclose the method 
according to Claim 1 1 , wherein a corresponding portion of said contextual data is 
obtained from a temporal region of interest for each source of said information. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image wherein a corresponding portion of contextual data is 
obtained from a temporal region of interest for each source of information (Figures 6-9; 
Column 6, Lines 30-67). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et al combination according to the 
teachings of Jain et al to obtain a corresponding portion of the contextual data from a 
temporal region of interest for each source of information because it will accurately 
classify and intelligently extract information, termed metadata, about the contents of 
video stream in real time (Jain et al, Column 1 , Lines 46-49). 

With regards to Claims 29 and 46, arguments analogous to those presented for 
Claim 12 are applicable to Claims 29 and 46. 
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Conclusion 



6. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 



7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Mehrdad Dastouri whose telephone number is (703) 
305-2438. The examiner can normally be reached on Monday to Friday from 8:00 a.m. 
to 4:30 p.m. If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Amelia Au can be reached on (703) 308-6604. 

The fax phone numbers for the organization where this application or proceeding 
is assigned are (703) 872-9306 for regular and for After Final communications. 
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